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DIGITAL PICTURE IMAGE COLOR COWERS ION 
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Field Of The Present Invention 

The present invention relates to color converting digital 
picture images, and more specifically to the correction of 
color imbalances while minimizing amplified noise. 

10 

Background Of Present Invention 

Digital color picture images acquired with a single 
sensor camera pass light through a color filter array (CFA) 
onto charge coupled device (CCD) , or other a sensor array. 

15 The sensor readings are digitized to form the electronic 

image. The spectral sensitivity of each pixel in a sensor 
array is typically adjusted by three sets of filters in a 
typical CFA. Commonly used filter sets include red R, green G 
and blue B color channels, or alternatively cyan, magenta and 

20 yellow color channels. Each pixel of the acquired picture 
area has an associated raw digital intensity value for each 
filter color channel. Similarly, a digital color picture 
image can be acquired using a multiple-sensor camera that uses 
three CCD's. 

25 Such digitized images will often be "out of balance" on a 

color display. One reason this occurs is the images are 
acquired under lighting conditions that affect the color 
intensity values recorded. A scene illuminated by tungsten 
lamps will have a yellow color cast. If the raw pixel values 

3 0 were simply displayed, the reproduced image would have a 
yellow cast. Human vision is tolerant of considerable 
variation in lighting color cast and compensates accordingly. 
Further color imbalance can result when the color channels in 
the CFA of a display device for displaying the image do not 

3 5 exactly match those initially used when the image was 
acquired. 

Color conversion processing is therefore often required 
to correct any color imbalance in an acquired image to be 



displayed. One common color conversion method calculates 
new pixel values by multiplying a matrix of raw pixel values 
with a color conversion matrix C nominal * That is, new channel 
values for each pixel can be calculated: 
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where Cnominal can be calculated by a least-square solution to 
minimize the sum of a squared-dif f erence between a spectral 
sensitivity function of the color-converted space and the 
standard color space. A suggested value for Cnominal is often 
provided by sensor manufacturers . The new red channel value 
Rnew, for example, is therefore a weighted sum of the raw red 
Rraw, green G ra w and blue B ra w color channel values. 

Each raw channel value includes inherent noise acquired 
during the image capture process from various noise sources. 
Possible noise sources include thermally generated readout 
noise, shot noise and fixed pattern noise. Thermally 
generated readout noise results from thermal agitation of 
electrons in the readout circuitry of the sensor. Shot noise 
results from the collision of photons upon impact with the 
sensor. Fixed pattern noise results from the inherent non- 
uniform characteristics of the sensor which are related to 
imperfections in the sensor manufacturing process. But 
multiplying the raw values by a color conversion matrix Cnominal 
effectively amplifies any inherent noise present, and often 
causes undesirable visual effects when the image is displayed 
after color conversion. 

There is a need for an alternative method of color 
converting a color picture image which simultaneously 
minimizes both the color error and amplified noise. 

Another color conversion application is in image/video 
compression. Since the RGB space is not an efficient space 
from compression viewpoint, R,G and B values are converted to 
YUV or YCbCr color space values before applying JPEG 



compression. The conversion from RGB space to YUV or YCbCr 
is thus. 
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Summary Of The Present Invention 

Briefly, a method embodiment of the present invention 
transforms the sensor color space of a digital image to 
standard color space. The color pixels in the sensor color 
space digital image are grouped and multiplied by multiple 
color conversion matrices. Different color conversion 
matrices are used for different parts of the image. The sum 
of the color error and amplified noise is minimized. 



Brief Description of the Drawings 

Fig. 1 is a diagram illustrating how an acquired picture 
area is divided into pixel groups according to an embodiment 
of the present invention; 

Fig. 2 is a flowchart diagram of a method embodiment of 
the present invention for color converting a picture image; 
and 

Fig. 3 is a functional block diagram of a digital camera 
and a processing system embodiment of the present invention. 



Detailed Description of the Embodiments 

Figs. 1A and IB represent a digitized picture area of an 
acquired color picture image 100, with 3200 pixels arranged as 
8x8 blocks 104. Each block 104 has an address- j . In Fig. 1A, 
ten such blocks 104 are in each row for 80 -pixels total, and 



five such blocks are in each column for 40-pixels total. 
Other configurations are possible, e.g., 1024x1024. Fig. IB 
details a single representative one of these 8x8 blocks 104. 
There are sixty-four pixels 102 in each 8x8 block 104. Each 
pixel 102 is assigned an address-i. 

Each 8x8 block 104 is allocated a pixel group position j 
ranging from 1 to the total number of pixel groups M. For the 
present example, j ranges from 1 to M, where M is fifty. Each 
pixel 102 in the 8x8 block 104 is allocated a pixel position i 
ranging from 1 to the total number of pixels N in the 8x8 
block 104. For the present example, i ranges from 1 to N, 
where N is sixty-four. 

Each pixel 102 has corresponding raw or unprocessed 
intensity values for its red R r aw> green G r aw and blue B raw 
channels. A matrix of sixty-four raw values for each color 
channel of an arbitrary 8x8 block 104 is shown in Table I. 



TABLE I 
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■NOMINAL 



3.7423 -0.9882 0.1377 
-2.1828 1.8432 -0.5416 
0.7365 -1.4519 1.4612 



R — 



9.9056 


10.2406 


10.2560 


10.0484 


9.9619 


9.6322 


10.1036 


9.6157 


11.0756 


10.6307 


10.8943 


9.8574 


9.0032 


8.8382 


9.1237 


8.8941 


10.0383 


10.1614 


9.2518 


9.2606 


9.9523 


10.0369 


10.0035 


10.0032 


10.5189 


9.3786 


10.1702 


9.1821 


10.2582 


9.4736 


9.4190 


9.2882 


10.7238 


10.4094 


9.8291 


9.1034 


8.3382 


9.1025 


10.2999 


8.8793 


9.9074 


9.8120 


10.4366 


10.1968 


9.8154 


9.1626 


9.4023 


9.2523 


9.4652 


9.8774 


9.6744 


9.0998 


9.2259 


8.3596 


9.4886 


8.8105 


8.1055 


6.7972 


6.9037 


7.1799 


7.2488 


7.8111 


6.4927 


6.6884 



G — 



29.2733 
28.3425 
28.4377 
27.5842 
27.9324 
27.9661 
27.2113 
20.6607 
30.9717 
31.6895 
30.0772 
29.6528 
30.5855 
28.9157 
29.2023 
25.0145 



27.8347 
28.8148 
27.6641 
26.6267 
27.7026 
26.8686 
26.0600 
20.9997 
30.4938 
30.0883 
31.0242 
28.5751 
29.5030 
29.7320 
28.1437 
24.5517 



27.9608 
28.1860 
26.5578 
26.9682 
26.9077 
26.1085 
25.1640 
19.2520 
30.5209 
28.6683 
29.8533 
27.9879 
29.0353 
29.5171 
27.0938 
22.5275 



26.3519 
26.6158 
27.3755 
25.9660 
28.2971 
26.0136 
24.7046 
18.9764 
28.5070 
28.0756 
29.4920 
28.8576 
29.7141 
29.0579 
26.9059 
23.2156 



25.4117 
25.8793 
26.8409 
26.1780 
26.3761 
26.2207 
23.5585 
19.6179 
27.6293 
27.3752 
28.1474 
29.1462 
28.2760 
27.4110 
26.2134 
22.2232 



25.3925 
25.9527 
25.3446 
25.9053 
26.5929 
25.9727 
22.6644 
19.0893 
27.6245 
26.1468 
27.2521 
27.3834 
28.8488 
27.4681 
25.1823 
22.8379 



24.4934 
25.0389 
24.4212 
24.8976 
25.7855 
23.8461 
23.4568 
18.7717 
26.9592 
26.8846 
26.8467 
28.2576 
28.0663 
26.9114 
26.3845 
21.7008 



24.3332 

25.1576 

25.7169 

24.9925 

24.8191 

22.3109 

23.2060 

18.2214 

25.3052" 

26.8315 

26.8692 

27.4453 

26.5923 

25.4566 

24.5598 

22.0221 
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Cor = 



89.0614 239.4668 260.4661 
239.4668 646.1170 702.3732 
260.4661 702.3732 764.5824 



Cor m = 



0.2043 0 0 
0 0.3033 0 
0 0 0.3168 



'NEW 



0.8839 0.0124 0.1923 
-0.2983 0.6309 -0.0697 
0.1104 -0.3452 0.6572 



12.4077 
17.8048 
13.6070 
16.1903 
16.7416 
13.4231 
12.5537 
13.3617 
10.0471 
13.3091 
10.0510 
11.0240 
12.0323 
8.9423 
10.1312 
12.5218 

15.5610 
10.9027 
14.2157 
11.8233 
11.5126 
14.2611 
13.6799 
6.8417 



15.0173 
15.4529 
14.9624 
12.7208 
15.6431 
14.2630 
15.0882 
8.0670 
11.6843 
9.9561 
12.6487 
9.9998 
10.5527 
11.6582 
10.5598 
10.3899 

12.4368 
13.6115 
12.0081 
13.1311 
12.3615 
12.0044 
11.2313 
10.5729 



14.9540 

16.8649 

12.4903 

15.2650 

14.1926 

17.3220 

15.0694 

9.9138 

11.5521 

8.9884 

11.8742 

9.2290 

10.5961 

12.9079 

10.1771 

10.0482 

12.6211 
12.6463 
12.5886 
12.3506 
12.4164 

9.3563 
10.5915 

8.2154 



15.4898 

14.4548 

11.6656 

12.6772 

10.1971 

16.4550 

13.3469 

11.3143 

10.7927 

9.6385 

10.1654 

11.2272 

9.0361 

12.1980 

10.1463 

11.6572 

11.1993 

12.3363 

14.2723 

12.1892 

16.1939 

9.9537 

11.1009 

6.7319 



15.9741 
11.8894 
14.5971 
16.5347 
9.0339 
14.5964 
14.8560 
10.8016 
10.8116 
9.0554 
9.4864 
12.1337 
9.1605 
9.2101 
10.8914 
9.3263 

10.1306 
13.2226 
12.5053 
10.0747 
15.1021 
12.0597 
9.0882 
8.3015 



14.7584 
11.0301 
16.2690 
13.6250 
11.7587 
12.4062 
12.3556 
13.5129 
10.5896 

7.0323 
10.4131 

9.3760 
10.2454 

9.1727 
10.0449 
11.4062 

10.8175 
14.3835 
10.0476 
12.2395 
13.5232 
12.9968 
9.8896 
5.7668 



17.3196 
13.1031 
17.0008 
14.5370 
16.9299 
15.3279 
15.9634 
8.7363 
11.2702 
9.6474 
11.1368 
12.0762 
11.1563 
11.6236 
11.4825 
9.2348 

8.4915 
11.6764 
8.6379 
10.0277 
9.8451 
8.8552 
8.2345 
8.6751 



15.4243 
12.1193 
15.7223 
13.8417 
12.3656 
16.0835 
13.4220 
10.0566 
8.7268 
9.2284 
9.2884 
10.6552 
9.3595 
11.6164 
8.6810 
10.6474 

10.1569" 

12.4252 

11.0146 

10.9282 

11.9629 

7.1407 
10.2408 

7.0597 



The red channel raw pixel value for a pixel position i in the 
10 8x8 block 104 is labeled R raw (i) ■ The value for i = 10 is 

therefore 10.6307. The color matrix of raw values 
corresponding to the pixel i = 10 can be represented as, 



"* raw (io)~ 
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B mw (10) 




30.0883 
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and can be substituted into Eq. 1 for color conversion of the 
tenth pixel where C NO minal is a suitable tristimulus conversion 
matrix as shown in Table I . 

A color conversion method embodiment of the present 
invention accounts for the sum of color error and noise in the 
raw values , which can be derived for each pixel i in each 
respective 8x8 block 104. 

Each raw color channel value in Eq. 1 includes noise, and 
therefore, 
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15 where N R , N G and N B are independent noise values in the red, 
green and blue color channels, respectively, and R, G and B 
are actual red, green and blue values, respectively. 

Considering Eq. 1 for a substantially noise-free system, 
the new noise- free green channel value G* new can be 

20 theoretically calculated: 



r 



R 
G 
B 



(3) 



where a* ,0* and y* are weights for the green channel in C NO minal , 
25 e.g., second row in C NO minal- 

In a noisy system, however, it is desirable to perform 
color conversion using a color conversion matrix C NBU of the 
present invention, wherein, 
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B. 



= C 



NEW 



R n 



(4) 



Analogous to Eq. 3, the green channel in a system subject 
to noise can be expressed as, 

G new =[a P Y. 

where, a.jffandy are weights for the green channel in the new 
color conversion matrix Chew (e.g., second row in C NEW ) . 



(5) 



Substituting Eq. 2 into Eq. 5 yields, 



R 



G new =[<* fi rjG + N, 
B + N 



B 



(6) 



C NEW can be derived so as to minimize the expected value f of 
the sum of color error and noise wherein, when considering the 
green channel, 



where E is the expected value. 



Substituting Eqs . 3 and 6 into Eq. 7 yields, 

f = El(a-a*)R + (/3-/3*)G + ( r -y t )B + aN R +/3N c +yN B ) 2 \ (8) 
= (a-a*) 2 E[R>]+(f}-f) 2 E[G 2 ] + ( r -/) 2 E[B 2 
+ 2(a-a*lp- f)E[RG] +2^-^-/ )e[GB[ 
+ 2(y -y'Xa -a)E[BR] + a 2 a 2 R + P 2 ct 2 g +r 2 <rl (9) 

where, » G and » s are estimated standard deviations of noise 

values N R , N G and N B respectively. 
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Eq. 9 can be minimized by taking partial derivatives 
with respect to or,/? and/ to yield, 



Cor 
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p 

r 



= (Cor-Cor NN ) 
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(10) 



where, Cor is the correlation matrix of [R raw G raw B raw f values 
and Cor NN is the correlation matrix of [N R N g N b Y values. 
Re-arranging Eq. 10 yields, 



= (Cor)~\Cor-Cor m ) 



a 

p' 

* 

r 



(ii) 



A similar derivation can be applied to obtain the weights 
of the red and blue channels, and combining them yields the 
new color conversion matrix 



CnEW =C N 0 M lNAL(C Or -C Or NN) T (C° r ~ 1 ) T < 12 > 



where Cor and Cor m can be estimated: 



Cor = — 
N 



Y R raMK aw ii) t, R raMG raw {i) 

1=1 /=! *=1 

iX»(0 G w(0 ZG raw (i).G raw {i) i]G raw (i>B raw (i) 

1=1 /=! 1=1 

f,K a MB raw {i) j^G^B^ii) JiBjtoBji) 



(13) 



Cor m = 



0 
0 



0 
0 



0 <T 



(14) 



In embodiments of the resent invention, Chew is based on 
actual pixel values, instead of Cmominal being a constant matrix 
based on the difference between color spaces. 

Substituting Eq. 12 into Eq. 4 for each respective 8x8 



block 104 yields. 



_ ^ new 



iff 
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NOMINAL y 
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5 where, i is the pixel position in the 8x8 block 104. 

Fig. 2 represents a method embodiment of the present 
invention, and is referred to herein by the reference numeral 
200. Method 200 can be implemented with software, for 
example, on a processing system 300 (Fig. 3). 

10 Fig. 3 represents a camera embodiment of the present 

invention, and is referred to herein by the reference numeral 
300. Camera 300 comprises a processing system 302 which 
includes an image memory 3 04, a microprocessor 3 06, and a 
program memory 308. Light from an image passes through a lens 

15 and color filter array 310, onto an image sensor array 312. 
Sensor readings are digitized by processor 3 06 to form a 
picture image 100 (Fig. 1) and is stored in the image memory 
3 04. The raw sensor color values acquired are color converted 
by processing system 302, e.g., according to method 200 (Fig. 

20 2) . 

Referring again to method 2 00 in Fig. 2, the processing 
is initiated with matrix C NO minal known and constant when 
converting each respective 8x8 block 104, see Table I. A step 
202 divides a picture image 100 (Fig. 1) into M multiple pixel 

25 groups 104 of N pixels 102. For the present example, M = 50 
and N = 64. A square block of sixty-four pixels will 
generally yield good results although this number can be 
varied. If N is too large, the method becomes less effective 
since C NE w cannot adapt to changes in the local signal 

30 (R,G,B) statistics. If N is too small, the accuracy of 

estimating Cor (eq. 13) is significantly degraded and the 
method is less effective. Thus, a compromise is generally 
required for the block size N owing to the statistical nature 
of the values used for performing the color conversion. 

35 Better conversion results are obtained if N is neither too 
small wherein not enough pixel values are considered for 



12 

estimating signal statistics, nor too large wherein the 
different signal statistics of various image regions are not 
properly accounted for. An 8x8 square block of sixty-four 
pixels provides a good compromise. 
5 A step 204 loads the raw pixel values of the red R r aw, 

green G raw and blue B raw channels for each pixel in the first 
8x8 block 104 (e.g., j = 1) , as indicated in Table I. A step 
206 calculates a correlation matrix Cor in accordance with Eq. 
13 for the position j of the current 8x8 block 104. The Cor 
10 for j = 1 is indicated in Table I. A step 208 calculates a 
correlation matrix Cor NN in accordance with Eq. 14 for a 
position j of the current 8x8 block 104. The Cor NN for j = 1 

is indicated in Table I. 

A step 210 calculates new pixel values for each pixel 

15 position i, in accordance with Eq. 15, where i varies from 1 
to N for a current 8x8 block 104. Table I indicates newly 
calculated pixel values for the red R n ew* green G new and blue 
Bnew channels for j = 1. A step 212 substitutes raw pixel 
values with a corresponding converted new pixel values 

20 calculated in step 210 for a current 8x8 block 104. 

A step 214 checks if the last 8x8 block 104 in a picture 
image 100 has been converted. If so, then the image 
conversion has been completed, and finishes as step 216. If 
not, the raw pixel values for the red R r aw, green G raw and blue 

25 B raw channels of a next-to-be-converted 8x8 block 104 are 

loaded in a step 218 by incrementing j (e.g., j = j+1) . Steps 
206 to 212 are repeated in a loop. 

The acquired picture image 100 stored in the image memory 
3 04 may be transferred from the digital camera 3 02 to a PC via 

3 0 a disk storage medium. The PC also contains a processing 

system 3 00 and can therefore be used to convert the picture 
image 100 after acquisition, according to method 200 (Fig. 2). 

Each block 104 can be other than 8x8. Although each 8x8 
block 104 in one embodiment is a square block of sixty-four 

35 pixels, this area can be rectangular, circular or even 

scattered wherein each 8x8 block 104 is effectively a set of 
pixels, the set comprising statistically similar pixels within 
the entire picture image 100 which are grouped according to 
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clustering or vector quantization algorithms. Each 

8x8 block 104 need not be of the same size when dividing a 
given picture image 100 and, in fact, pixel groups 104 can 
overlap or may be displaced such that some areas of the 
picture image 100 are not color converted at all. 

Being able to calculate the inverse of the correlation 
matrix Cor in Eq. 15 is sometimes difficult to implement. The 
computational complexity of this calculation may be reduced by 
using alternative numerical algorithms, e.g., conjugate 
gradient or steepest descent methods, wherein the initial 
starting point for C NEW is either C NOMINAL or the C NEW matrix of 

an adjacent 8x8 block 104. 

According to another embodiment of the present invention, 
there may be more than three color channels, and therefore 
more than three corresponding values, for each pixel in the 
picture image 100. The present invention is not to be limited 
to converting only three color channels of the RGB color 
space. In another embodiment of the present invention, 
different color channels may be used such as cyan, magenta and 
yellow. Alternatively, cyan, magenta, yellow and black (CMYK) 
or cyan, magenta, yellow and white could be used. 

In a further an embodiment of the present invention, the 
conversion method could be applied to the entire picture image 
100. For larger images, edges could be detected before, and 
subsequently re-inserted after color conversion processing is 
undertaken to minimize blur at the edges. 

Further, one embodiment involved processing raw pixel 
values, however, the same color conversion technique can be 
applied to the pixel values of a compressed image such as a 
JPEG or like compression processed image file. In addition, 
the processing may be done after the picture image 100 is 
acquired. In other words, the image may be acquired and 
stored, and then subsequently retrieved and processed using 
the aforementioned technique. 

Each color channel has different noise statistics and 
C NEW is adaptively weighted depending on the noise 

characteristics of the 8x8 block 104. According to another 
embodiment of the present invention, the weights for a given 
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color channel could be manually selected and inserted 

into C NEW for particularly noisy channels. 

These and other variations and embodiments should be 
considered to fall within the scope of the invention 
5 disclosed. 



